26 research outputs found
Efficient Quantization-aware Training with Adaptive Coreset Selection
The expanding model size and computation of deep neural networks (DNNs) have
increased the demand for efficient model deployment methods. Quantization-aware
training (QAT) is a representative model compression method to leverage
redundancy in weights and activations. However, most existing QAT methods
require end-to-end training on the entire dataset, which suffers from long
training time and high energy costs. Coreset selection, aiming to improve data
efficiency utilizing the redundancy of training data, has also been widely used
for efficient training. In this work, we propose a new angle through the
coreset selection to improve the training efficiency of quantization-aware
training. Based on the characteristics of QAT, we propose two metrics: error
vector score and disagreement score, to quantify the importance of each sample
during training. Guided by these two metrics of importance, we proposed a
quantization-aware adaptive coreset selection (ACS) method to select the data
for the current training epoch. We evaluate our method on various networks
(ResNet-18, MobileNetV2), datasets(CIFAR-100, ImageNet-1K), and under different
quantization settings. Compared with previous coreset selection methods, our
method significantly improves QAT performance with different dataset fractions.
Our method can achieve an accuracy of 68.39% of 4-bit quantized ResNet-18 on
the ImageNet-1K dataset with only a 10% subset, which has an absolute gain of
4.24% compared to the baseline.Comment: Code: https://github.com/HuangOwen/QAT-AC
LLM-FP4: 4-Bit Floating-Point Quantized Transformers
We propose LLM-FP4 for quantizing both weights and activations in large
language models (LLMs) down to 4-bit floating-point values, in a post-training
manner. Existing post-training quantization (PTQ) solutions are primarily
integer-based and struggle with bit widths below 8 bits. Compared to integer
quantization, floating-point (FP) quantization is more flexible and can better
handle long-tail or bell-shaped distributions, and it has emerged as a default
choice in many hardware platforms. One characteristic of FP quantization is
that its performance largely depends on the choice of exponent bits and
clipping range. In this regard, we construct a strong FP-PTQ baseline by
searching for the optimal quantization parameters. Furthermore, we observe a
high inter-channel variance and low intra-channel variance pattern in
activation distributions, which adds activation quantization difficulty. We
recognize this pattern to be consistent across a spectrum of transformer models
designed for diverse tasks, such as LLMs, BERT, and Vision Transformer models.
To tackle this, we propose per-channel activation quantization and show that
these additional scaling factors can be reparameterized as exponential biases
of weights, incurring a negligible cost. Our method, for the first time, can
quantize both weights and activations in the LLaMA-13B to only 4-bit and
achieves an average score of 63.1 on the common sense zero-shot reasoning
tasks, which is only 5.8 lower than the full-precision model, significantly
outperforming the previous state-of-the-art by 12.7 points. Code is available
at: https://github.com/nbasyl/LLM-FP4.Comment: EMNLP 2023 Main Conferenc
The role of cancer-associated fibroblasts in breast cancer metastasis
Breast cancer deaths are primarily caused by metastasis. There are several treatment options that can be used to treat breast cancer. There are, however, a limited number of treatments that can either prevent or inhibit the spread of breast tumor metastases. Thus, novel therapeutic strategies are needed. Studies have increasingly focused on the importance of the tumor microenvironment (TME) in metastasis of breast cancer. As the most abundant cells in the TME, cancer-associated fibroblasts (CAFs) play important roles in cancer pathogenesis. They can remodel the structure of the extracellular matrix (ECM) and engage in crosstalk with cancer cells or other stroma cells by secreting growth factors, cytokines, and chemokines, as well as components of the ECM, which assist the tumor cells to invade through the TME and cause distant metastasis. Clinically, CAFs not only foster the initiation, growth, angiogenesis, invasion, and metastasis of breast cancer but also serve as biomarkers for diagnosis, therapy, and prediction of prognosis. In this review, we summarize the biological characteristics and subtypes of CAFs and their functions in breast cancer metastasis, focusing on their important roles in the diagnosis, prognosis, and treatment of breast cancer. Recent studies suggest that CAFs are vital partners of breast cancer cells that assist metastasis and may represent ideal targets for prevention and treatment of breast cancer metastasis
Improvements in the preparation of phosphate for oxygen isotope analysis from soils and sediments.
In contrast to the successful preparation of phosphate for oxygen isotope analysis from water samples, there are still a series of problems for similar analyses from soils and sediments. Here, we improved and optimized the methods of silver phosphate preparation for oxygen isotope analysis from soils and sediments. During our preparations, organic matter was removed by sodium hypochlorite and XAD-2 resin, while the impurities of elemental silver and its oxide were removed by rapid microprecipitation and ammonium phospho-molybdate and magnesium ammonium phosphate. The total organic carbon and total nitrogen in the prepared silver phosphates from soils and sediments were 0.226±0.033% and 0.030±0.0059% (n = 7), 0.217±0.053% and 0.034±0.0120% (n = 9), respectively, indicating a high removal efficiency of organic matter. We confirmed that adding citric acid during rapid microprecipitation would introduce the impurity of elemental silver, which could be removed by ammonia recrystallization. The pH range of solutions for rapid microprecipitation was optimized at 7.0‒7.5. Results of X-ray Diffraction and stable oxygen isotope analyses showed that the improved method could obtain high pure silver phosphate from soil and sediment samples without oxygen isotope fractionation. This improved procedure provides a foundation for biogeochemical studies on phosphorus in soil and lacustrine environments by using phosphate oxygen isotopes